• 专利附录
  • 专利说明
  • 权利要求
  • 类似技术
  • 同族专利
  • 引用文献
  • 法律状态
  • 优先权
    • 专利摘要:
    The method comprises for each current pair (PPi) of first (IM1) and second (IM2) successive video images, a motion determination between these two images comprising a test phase of several hypotheses of homographic models of said motion by an algorithm of RANSAC type operating on a set of first points (Plj) of the first image and corresponding first presumed points (P2j) of the second image so as to deliver the best homographic model hypothesis, this best homographic model hypothesis defining said motion. Said test phase comprises a test of several first hypotheses of homographic models (Hlk) of said movement obtained from a set of second points (Plj) of the first image and second corresponding presumed points (P2j) of the second image and at least one second homographic model hypothesis (H2) obtained from auxiliary information (Ox, Oy, Oz) provided by at least one inertial sensor (30) and representative of a movement of the image sensor between the captures the two successive images (IM1, IM2) of said pair.
    公开号:FR3027144A1
    申请号:FR1459675
    申请日:2014-10-09
    公开日:2016-04-15
    发明作者:Manu Alibay；Stephane Auberger
    申请人:Association pour la Recherche et le Developpement des Methodes et Processus Industriels；STMicroelectronics SA；
    IPC主号:
    专利说明:

    [0001] Method and device for determining motion between successive video images Embodiments of the invention relate to the motion determination between successive video images captured by an image sensor, for example a video camera. such as for example that incorporated in a digital tablet or cellular mobile phone. Video image sequences can have many quality problems, especially when they are processed by embedded processors, such as those present in digital tablets or cellular mobile phones. Among these quality problems, one can cite the presence of fuzzy contents, unstable contents, or deformations due to the so-called "rolling shutter" effect which induces a deformation of the images acquired during a movement of the camera due that the acquisition of an image via a CMOS sensor is done sequentially line by line and not all at once. All these problems are due to the movement between successive images. It is therefore necessary to make an estimate. It is known to estimate the global movement between two successive video images by a homographic model, typically a 3x3 homography matrix modeling a global plane motion. Typically, homographic matrices are estimated between successive images using feature mappings between these images. Algorithms for estimating such matrices between successive images are well known to those skilled in the art and it may for all intents and purposes refer to Elan Dubrofsky's essay entitled "Homography Estimation", B. Sc. , Carleton University, 2007, THE UNIVERSITY OF BRITISH COLUMBIA (Vancouver), March 2009. The RANSAC algorithm, abbreviation of "Random Sample Consensus", well known to those skilled in the art and described in particular in the Fischler article 3027144 2 et al., entitled "Random Sample Consensus: A Paradigm for Modeling with Applications to Image Analysis and Automated Cartography," communications of the ACM, June 1981, Volume 24, No. 6, is a robust algorithm for estimating 5 parameters used in particular in image processing applications, to estimate the overall motion between two images by testing a number of homographic models. More specifically, in a first step, one randomly selected from all available points (pixels) of a current image, a generally minimal set of points of the current image, for example a triplet of points. We extract the corresponding presumed point triplet in the following image and we estimate from these two triplets a homographic matrix representing a motion model hypothesis.
    [0002] This model hypothesis thus obtained is then tested on the complete set of points of the image. More specifically, for at least some of the points of the image, an estimated point is calculated using the tested model hypothesis and the backprojection error is determined between this estimated point and the corresponding presumed point in the next image. . The points that do not follow the model, that is to say, whose overhead projection error is greater than a threshold T, are called "outliers". On the other hand, the points close to the model hypothesis are called "inliers" and are part of the "consensus set". Their number is representative of the quality of the estimated model hypothesis. The two previous steps (choice of a model hypothesis and test on all the points) are repeated as long as the number of iterations has not reached a threshold defined by a formula taking into account the percentage "d". desired inliers and a desired confidence value. When this condition is satisfied, the model hypothesis that led to this condition is then considered to be the model of the overall motion between the two images.
    [0003] However, the calculation time of the RANSAC type algorithm is very variable and depends in particular on the number of points tested and the quality of the points. Indeed, in an image called "easy", having in particular many points of interest characteristic of the image, one will easily find the corresponding presumed points in the following image but this will not be the case in an image called " difficult ". This variability of the calculation time is generally not compatible with the use of such an algorithm in embedded processors for example within cellular mobile phones or digital tablets. Also, in such embedded applications, a preemptive RANSAC type algorithm (Preemptive RANSAC) well known to those skilled in the art and preferably described in the article by David Nister, entitled "Preemptive RANSAC for Live Structure", is preferably used. Motion Estimation, Proceedings of the Ninth IEEE International Conference on Computer Vision (ICCV 2003) 2 Volume Set. In the preemptive RANSAC algorithm, one defines beforehand 20 from a set of points of the current image (called "set of hypothesis generating points") and their correspondences in the preceding image, a set of K homographic models, which constitutes K hypotheses of models to be tested. Typically, K may be between 300 and 500.
    [0004] Then, all these models are tested, in a manner analogous to that performed in the conventional RANSAC algorithm, on a first block of points of the image, for example 20 points. At the end of this test, only a part of the tested model assumptions, typically those that achieved the highest scores, are retained. One can for example perform a dichotomy, that is to say, to keep only half of the assumptions of models tested. Then, the remaining model hypotheses are tested using another block of points and, again, for example, only half of the hypotheses of tested models having obtained the highest scores are retained. These operations are repeated until all the points are exhausted or a single model hypothesis is finally obtained.
    [0005] In the latter case, this unique remaining model hypothesis forms the overall model of motion between the two images. In the case where there are several hypotheses of models but more points to be tested, the hypothesis retained is that presenting the best score. That is, even if the preemptive RANSAC algorithm has certain advantages in particular in terms of computing time, which makes it particularly well suited for embedded applications, and furthermore for parallel processing, the motion estimation is less flexible and sometimes not really suitable for extreme cases. Thus, for example, if a person or object moves in the field of an image, it may happen that the motion estimator focuses on the person, producing a result that does not correspond to the movement of the camera, this which could for example provide an erroneous video stabilization. According to one embodiment and embodiment, it is proposed to improve the motion estimation between successive video images allowing an improvement in the quality of the image sequence, in particular in certain particular situations. In one aspect, there is provided a method of determining motion between successive video images captured by an image sensor, the method comprising for each current pair of first and second successive video images (the first and second successive video images a pair can typically be the previous image and the current image), a motion determination between these two images, this motion determination comprising a test phase of several hypotheses of homographic models of said motion by an algorithm of the type RANSAC operating on a set of first points (test) of the first image and first points (test) 3027144 5 alleged corresponding of the second image so as to deliver the best hypothesis of homographic models, this best hypothesis of homographic models defining said movement. According to a general characteristic of this aspect, the test phase comprises a test of several first hypotheses of homographic models of said movement obtained from a set of second points of the first image and second corresponding presumed points of the second image (points hypothesis generators), and at least one second homographic model hypothesis obtained from auxiliary information provided by at least one inertial sensor and representative of a movement of said image sensor between the captures of the two successive images. of said pair. Thus, according to this aspect, information from at least one inertial sensor, for example at least one gyroscope, is used in combination with the visual information to improve the estimation of motion between two successive images. And, for that, we add in the hypotheses of models tested by the RANSAC type algorithm, a hypothesis of homographic model that can be described as "inertial" and which is directly determined from the information provided by the or the inertial sensors which are representative of a movement of said image sensor between the captures of the two successive images. Thus, for example, the inertial sensor (s) may be incorporated into the cellular mobile phone or tablet which also incorporates the image sensor. Further, the testing of each homographic model hypothesis will advantageously take into account a distance between said homographic model hypothesis tested and said at least one second homographic model hypothesis (i.e., the homographic model hypothesis). "Inertial"). In fact, each model hypothesis, whether it be a first model hypothesis (that is, a "visual" model hypothesis) or the second model hypothesis ("inertial"), is advantageously same way by the RANSAC type algorithm. In other words, as will be seen in more detail below, a score will be assigned to each model assumption and this score will be advantageously corrected taking into account said distance. Of course, when it comes to testing the second model hypothesis (the inertial model hypothesis), said distance is zero, and therefore the score obtained by this inertial model on the basis of the correspondence of points between the first image and the second image is corrected with zero correction which amounts to not correcting it. Although it is possible to use a conventional RANSAC-type algorithm, it is particularly advantageous to use a preemptive RANSAC-type algorithm, especially for embedded applications having constraints in terms of computing time. The set of first points (test) on which we will apply the test phase of the RANSAC type algorithm and the set of second points (hypothesis generators) from which we determine the first model assumptions 20 homographic, may or may not overlap. The points of these two sets may advantageously be points of interest of this image, that is to say easily recognizable characteristic points from one image to another.
    [0006] In practice, according to an implementation mode in which the preemptive RANSAC type algorithm is applied, the set of second points randomly draws triplets to generate a number of homography candidates (hypotheses). and organizing all the first (test) points grouped by "blocks" which will be progressively tested by the preemptive RANSAC type algorithm. As indicated above, the set of second points (hypothesis generators) may or may not overlap with the set of test points, these sets of points being advantageously obtained by random draws.
    [0007] Although it may be sufficient to use only a gyroscope to provide auxiliary information of the inertial type, it may be preferable to use in addition to a gyroscope at least one other inertial sensor taken from the group formed by one or more accelerometers and a magnetometer. The accelerometer may be a three-axis accelerometer or three accelerometers placed along perpendicular axes. The accelerometer can thus give an indication of the gravity while the magnetometer makes it possible to obtain an orientation indication since it provides an estimated direction of the north. According to one mode of implementation, the test of each homographic model hypothesis, whether it be a first homographic model hypothesis ("visual" hypothesis) or a second hypothesis of a homographic model ("inertial" hypothesis), includes for each first point of at least one block of said set of first points of the first image - a determination of a first estimated point in said second image from the hypothesis of a homographic model tested, - a determination of a difference of position between the first estimated point and said first corresponding presumed point in said second image, - a determination of a first score information from the positional deviations obtained and an error tolerance, and - a correction of said first score information with a correction element having a first coefficient taking into account the distance between the model hypothesis tested and the second model hypothesis, so as to obtain a second score information, this second score information being used for the determination of said best homographic model hypothesis. The determination of said corrective element may also include a weighting of said first coefficient by a weighting coefficient representative of a weight of the first score information associated with said at least one second homographic model hypothesis with respect to the first score information. associated with said homographic model hypothesis tested. This weighting coefficient may have a fixed value and the same for all hypotheses of homographic models tested of all the pairs of images. If the value of this weighting coefficient is too high, then we will give too much importance to the inertial model but not enough importance if the weighting coefficient is too low.
    [0008] A fixed and constant value equal to 1 may be a good compromise. That being so, alternatively, the weighting coefficient may have a fixed and identical value for all tested homographic model assumptions of said current pair of images, but this value may be calculated at each new current pair of images. And, the calculation of this value can be done from all the values of respective distances between the hypotheses of homographic models tested of the current pair of images and the second hypothesis of homographic models.
    [0009] The determination of said corrective element can also take into account the number of second points. In another aspect, there is provided a motion determining device between successive video images, comprising input means configured to receive image signals relating to video images successively captured by an image sensor, processing means configured to perform, for each current pair of first and second successive video images, a motion determination between these two images, the processing means comprising test means configured to test several homographic model assumptions of said motion by a RANSAC type algorithm operating on a set of first points of the first image and corresponding first presumed points of the second image so as to deliver the best homographic model hypothesis, this best homographic model hypothesis defining said motion. According to a general characteristic of this other aspect, the device comprises auxiliary input means configured to receive auxiliary information from at least one inertial sensor and representative of a movement of said image sensor between the captures of the two successive images. of said pair, the test means being configured to test several first hypotheses of homographic models of said movement obtained from a set of second points of the first image and second corresponding presumed points of the second image and at least a second homographic model hypothesis obtained from said auxiliary information. According to one embodiment, the auxiliary input means 15 are configured to receive said auxiliary information from at least one gyroscope. According to another possible embodiment, the auxiliary input means are configured to receive said auxiliary information from a gyroscope and at least one other sensor taken from the group formed by one or more accelerometers and a magnetometer. The RANSAC type algorithm can be a preemptive RANSAC type algorithm. According to one embodiment, the test means are further configured to, when testing each homographic model hypothesis, take into account a distance between said homographic model hypothesis and said at least one second homographic model hypothesis. According to one embodiment, the test means comprise a test module configured to test each tested homographic model hypothesis, said test module comprising first determination means configured to determine, for each first point of at least one block said set of first points of the first image, a first point estimated in said second image from the tested homographic model hypothesis, second determining means configured to determine a positional difference between the estimated first point and said The first corresponding presumed point in the second image, third determination means configured to determine a first score information from the positional deviations obtained and an error tolerance, calculation means configured to calculate a corrective element. having a first coefficient taking into account said distance, e correction means configured to perform a correction of said first score information with said corrective element so as to obtain a second score information, this second score information being used for the determination of said best homographic model hypothesis. According to one embodiment, the calculation means are further configured to weight said first coefficient by a weighting coefficient representative of a weight of the first score information associated with said at least one second homographic model hypothesis by report to the first score information associated with said homographic model hypothesis tested.
    [0010] According to one embodiment, the weighting coefficient has a fixed and identical value for all hypotheses of homographic models tested of all the pairs of images. According to another possible embodiment, the weighting coefficient has a fixed and identical value for all hypotheses of homographic models tested of said current pair of images, the calculation means being configured to calculate this value from all respective distance values between the tested homographic model assumptions of said current image pair and the second homographic model assumption, and for recalculating this value to each new current pair of images. The calculation means are further advantageously configured to also take into account the number of second points.
    [0011] In another aspect, there is provided a processing unit, for example a microprocessor or a microcontroller, incorporating a motion determining device as defined above. According to another aspect, there is provided an apparatus, for example a cellular mobile telephone or a digital tablet, incorporating an image sensor, at least one inertial sensor, and a processing unit as defined above, coupled to said image sensor and auditing at least one inertial sensor so as to receive said image signals and said auxiliary information. Other advantages and characteristics of the invention will become apparent on examining the detailed description of embodiments and embodiments, in no way limiting, and the accompanying drawings, in which: FIGS. 1 to 5 schematically illustrate the modes; implementation and realization of the invention.
    [0012] In FIG. 1, reference APP denotes an apparatus such as for example a cellular mobile telephone or a digital tablet, without these examples being limiting, comprising an image sensor 2, for example an on-board camera, intended to film an SC scene.
    [0013] The apparatus APP also comprises a device 1 for determining motion between successive video images captured by the image sensor 2. This device 1 may for example be incorporated within a microprocessor. The device 1 comprises input means 10 for receiving image signals relating to the video images of the scene SC successively captured by the image sensor 2 and auxiliary input means 11 intended to receive auxiliary information. for example from a gyroscope 30 and possibly one or more accelerometers 31 and / or a magnetometer 32.
    [0014] The inertial sensors 30, 31 and 32 are for example integral with the APP apparatus in the same way as the image sensor. The inertial sensors consequently follow the possible movement in the space of the image sensor 2.
    [0015] As a result, this auxiliary information is representative of a movement of the image sensor between the captures of two successive video images. The device 1 comprises processing means 10 configured, as will be seen in more detail below, to perform, for each current pair of first and second successive video images, a motion determination between these two images. In this regard, the processing means comprise test means 100 configured to test several first hypotheses of homographic models of this movement, obtained from a set of points of the first image and corresponding presumed points of the second image. , and at least one second homographic model hypothesis obtained from the auxiliary information. The test module 100 comprises in this respect various means 20 referenced 1001-1005 which will be discussed in more detail below on the function. Materially, the processing means 10 and the means that compose it can be implemented in a software manner within the microprocessor.
    [0016] The different hypotheses of homographic models of the movement will be processed by a RANSAC type algorithm. And, although the conventional RANSAC type algorithm can be used, an embodiment of the invention will now be described using the preemptive RANSAC type algorithm better adapted for embedded applications, such as is the case described here with reference to FIG. 1. In general, the preemptive RANSAC algorithm operates on successive blocks of a set of first points of the first image and corresponding first presumed points of the second 3027144 13 image of a pair and tests in particular hypotheses of homographic models, called "visual", obtained from a set of second points of the first image and second corresponding presumed points of the second image.
    [0017] 5 That being so, in general, these second points are points of interest of the image and the set of first points (of tests) can intersect or not with the set of second points (hypothesis generators). Reference will now be made more particularly to FIG. 2 to describe an embodiment of the motion determination method between successive video images. In step 20, successive video images are captured and it is assumed here that a first video image IM1 and a second video image IM2 of a current pair of PPS images have been captured.
    [0018] The first image IM1 is typically the previous image and the second image IM2 is the current image. An extraction in the first image IM1 of N points or pixels P1, j = 1 to N, and an extraction of N points or corresponding presumed pixels P2i in the second image IM2 are then carried out.
    [0019] This extraction of points of interest in an image and corresponding presumed points in the next video image can be performed with any algorithms known to those skilled in the art, for example the algorithm known to those skilled in the art under 'Acronym 25' Fast 'and described for example in the article by Edward Rosten and Tom Drummond entitled' Machine learning for high-speed corner detection ', ECCV'06 Proceedings of the 9th European Conference on Computer Vision, Volume 1, Part 1 , pages 430-443, and the algorithm known to those skilled in the art under the acronym "Brief" and described for example in the article by Michael Calonder and others entitled "Brief: binary robust independent elementary features", ECCV Proceedings of the Third European Conference on Computer Vision: Part IV, pages 778-792.
    [0020] From these points P1 and P2i, we form triplets of points of the first image and triplets of corresponding presumed points of the second image, and from these triplets (step 25) K first assumptions are developed. from homographic models Hlk, k = 1 to K, of the overall motion between the two images IM1 and IM2. These first hypotheses of homographic models are 3x3 homography matrices obtained, for example, using the Direct Linear Transform (DLT) algorithm described for example in the aforementioned Elan Dubrofsky test.
    [0021] These first hypotheses of models Hlk can be considered as hypotheses of "visual" models since they are obtained from the pixels of the two successive images IM1 and IM2. As an indication, the number K of the first hypotheses of 15 Hlk models can be between 300 and 500. Furthermore, the processing means 10 will develop from the auxiliary information provided by the gyroscope 30, and possibly the accelerometer or accelerometers 31 and / or magnetometers 32, a second homographic model hypothesis H2 that can be referred to as the "inertial" model hypothesis being that it is obtained directly from the auxiliary information delivered by the inertial sensor (s). . The types of cellular mobile phones known as "smartphone" may be equipped with 25 gyroscope, accelerometer and magnetometer. It is the same for the current digital tablets. It is assumed here that only a gyroscope is present. The gyroscope integrates the rotational speeds on the three axes between the capture of the two images and provides the auxiliary information 30 ex, ey and Oz which are the corresponding angles of rotation about the x, y and z axes and which respectively represent the yaw ( "Yaw") pitch (pitch) and roll ("roll"). In order to develop the 3 × 3 homerographic inertial matrix, the processing means must determine the horizontal translation 4T ,, and vertical 4Ty as well as the angle of rotation in the plane resulting from the movement of the sensors between the two captured images IM1 and IM2. In this regard, 4Tx is given by formula (1) below: AT '= 0' - p '(1) wherein Px is a scale factor defined by formula (2) below: px = Lx / 2tan-1 (4 / 2fx) (2) Similarly, 4Ty is defined by the formula (3) below: ATy = y-py (3) in which py is a defined scale factor by formula (4) below: py = Ly 12 tan -1 (Ly / 2fy) (4) In formulas (2) and (4) Lx and Ly represent the resolution of the image, fx and fy the focal length and x and y refer respectively to the horizontal and vertical directions of the image. The use of such scale factors is well known to those skilled in the art and it may for all intents and purposes be referred to the article by Suya You et al. Entitled "Hybrid inertial and vision tracking for augmented reality registration". Virtual Reality, 1999, Proceedings, IEEE 13-17 March 1999 pages 260-267. The roll angle Oz directly provides the planar rotation angle without the need for a scale factor.
    [0022] The second homographic model hypothesis ("inertial") can then be represented by the 3 × 3 H 2 matrix defined by the formula (5) below: ## EQU1 ## , (5) 0 0 1 If the telephone is also equipped with an accelerometer (s) and / or a magnetometer, the information provided by the gyroscope is corrected, for example by filtering, in a manner known to provide said auxiliary information. From here, calculation means 1004 (FIG. 1) belonging to the test module 100 calculate (step 24, FIG. 2) a corrective element CORRk for each first model hypothesis Hlk taking into account a distance between said hypothesis Hlk. and the second hypothesis H2 .. The calculation of these correction coefficients will be discussed in more detail below: The test module 100 will then proceed to test the various hypotheses of homographic models, in this case the first hypotheses H lk and the second hypothesis H2.For this, since the preemptive RANSAC type algorithm is used, the test module randomly extracts from the set of points P1 a block of test points BL1Ai and extracted from the set Of the points P2j the block of corresponding presumed points BL2Ai, with i varying from 1 to I. By way of indication, I may be equal to 20. In other words, the first hypothesis is to test the different hypotheses. homographic models on a random 20-point block in image IM1 and on the block of corresponding presumed points in image IM2. At least some of these test points may or may not be taken from among the points used to develop the different model assumptions. To perform this test, first determination means 1001 (FIG. 1) determine, for each point BL1Ai of the block, an estimated point BL1ASi in the second image IM2, using the tested homographic model hypothesis.
    [0023] Secondly, second determination means 1002 (FIG. 1) determine the positional deviation ei, k between the estimated point BL1AS and the corresponding presumed point in the second image BL2A. As an indication, this positional deviation ei, k which corresponds to the number of pixels between the two points can be normalized according to formula (6) below: = BL1AS - (6) 10 in which the notation 11 11 represents the standard function. In addition, third determination means 1003 (FIG. 1) are configured to determine a first score information SCV1k for the first hypotheses Hlk and a first score information SCV2 for the second homographic model hypothesis H2, based on the differences of position obtained and an error tolerance. More precisely, at the beginning of the test, the score information SCV1k and SCV2 are initialized to 0. And, whenever the positional deviation ei, k (for i = 1 to I) associated with a hypothesis Hlk is greater than at a predefined error ERR, the corresponding score information SCV1k remains unchanged while it is updated by the formula (7): SCVL, = SCVL, +1 (7) 25 if the position deviation ek is lower or equal to said ERR error. The updating of the SCV2 score information associated with the H2 model assumption is performed in an identical manner.
    [0024] Once the I points BL1Ai have been processed, one thus obtains for each first homographic model hypothesis Hlk and for the second homographic model hypothesis H2 the first updated score information SCV1k and SCV2 that can be qualified here as "visual" score information since they were obtained using the points contained in the two images IM1 and IM2. Then, the correction means 1005 (FIG. 1) make a correction of this visual score information using the correction coefficients CORRk. More precisely, for each first model hypothesis Hlk, a second score information SCV1Ck obtained by the following formula (8) is obtained: SCV1Ck = SCV1k - CORRk (8) As for the second score information SCV2C associated with the Inertial model hypothesis H2, it is simply equal to the corresponding visual score information SCV2 since the correction coefficient applied to it is zero. In a next step 28, the test module performs, for example, a dichotomy on the model hypotheses Hlk and H2 that have just been tested. Specifically, the test module retains only half of the 20 tested model hypotheses that had the second highest scoring information. Then, the test module again performs a test 29 on these remaining model assumptions using a new block of BL113 points, of the first frame, drawn randomly from the points not already tested, and the corresponding presumed block of points. BL213, of the second image IM2. The operations that have just been carried out are repeated either until a single hypothesis of the remaining model HF is obtained, or until the tested points are exhausted.
    [0025] In the first case, the remaining model hypothesis HF then represents the global motion model between the two images IM1 and IM2. In the second case, we will retain as the HF hypothesis, the one with the second highest score information.
    [0026] More particularly, reference will now be made to FIGS. 3 to 5 to describe in more detail examples of calculation of the correction coefficients CORRk. More precisely, referring firstly to FIG. 3, it can be seen that the calculation means 1004 (FIG. 1) first calculate (step 240) the distance between the first homographic model hypothesis considered Hlk and the second homographic model hypothesis H2. It is recalled here that the first homographic model hypothesis Hlk is a 3x3 matrix as defined by the following expression (9): (a1 a2 a3 H1k = a4 a5 a6 (9) a7 a8 a9 The matrix H2 is that illustrated by the formula (5) above The two matrices being of the same structure, the coefficients a3 and a6 of the matrix Hlk respectively represent translations in x and in y while the coefficient a2 represents the sine of the angle of rotation in the plane.
    [0027] Accordingly, a particularly simple way of determining the distance dk (Hlk, H2) between the two model assumptions is to use formula (10) below: dk (Hlk, H2) = ka3 -AT x) 2 ± (a6 - ATy (arcsin (a2) - Oz) 2/2 (10) It is of course noted that the distance d (H2, H2) is obviously nil, the calculation means then determine (step 241) a first coefficient (clk) defined by the formula (11) below: in which e denotes the exponential function, of course, the first coefficient associated with the second hypothesis of model H2. The calculating means then determines (step 242) a weighting coefficient X representative of a weight of the score information associated with said second homographic model hypothesis H2 with respect to the score information associated with the hypothesis of homographic model tested Hlk or H2.We will return in more detail below on how to determine this weighting coefficient. The calculating means then determine (step 243) the corrective element CORRk by the following formula (12): CORRk = N.2 .C1k (12) 15 in which N designates the number of points tested, that is, say the number of points Pli and the number of points P2i (j = 1 to N), (figure 2). The lower the X weighting coefficient, the greater the visual score of the first model hypotheses Hlk will have weight compared to the inertial score of the hypothesis of inertial model H2. On the other hand, the higher the X weighting coefficient, the less the visual score of the first model hypotheses Hlk will have weight compared to the inertial score of the inertial model hypothesis 25 H2. Those skilled in the art will be able to determine the weighting coefficient X as a function of the intended application. That being so, a fixed and constant value X equal to 1 for all homographic model assumptions is a good compromise.
    [0028] It is quite possible to keep this fixed and constant value X equally for all successive image pairs. However, as a variant, in order to further improve the quality of the video sequence filmed, it is possible, as illustrated schematically in FIG. 4, for the determination 242 of the weighting coefficient X to be recalculated for each pair PPp of FIG. successive images IM1p, IM2p. More specifically, for each current pair of images PPp, the calculation means determine (step 2420) the value of the weighting coefficient Xp, which will remain the same for all the tested model hypotheses associated with these two images of the current pair. PPp. An example of calculation of the weighting coefficient X.p is illustrated in FIG.
    [0029] More precisely, the calculation means calculate in a step S10 all the distances dk (Hlk, H2) for k = 1 to K, between the first assumptions of models H1 and the second hypothesis of model H2. Then, the calculation means calculate in step S20 the first 15 corresponding coefficients clk (see step 241 of FIG. 3). The calculation means then extract from all the first coefficients clk the median coefficient referenced cm (step S30). The weighting coefficient Xp is then defined by the formula 20 (13) below: 2 = 2e '(13) Such a variable coefficient Xp between the different images 25 makes it possible not to override the internal movement of an object within the image in relation to the background. Thus, for example, when a truck crosses the camera field and occupies almost all this field, it minimizes the stabilization of the image on the truck and thus minimizes the movement of the background.
    [0030] In its generality, the invention also makes it possible, for example when filming a black dot in the center of a white wall, to have only a slight oscillation of the black dot due to the inaccuracy of the inertial sensor (s).
    权利要求:
    Claims (22)
    [0001]
    REVENDICATIONS1. A method of determining motion between successive video images captured by an image sensor, comprising for each current pair (PPS) of first (IM1) and second (IM2) successive video images, a motion determination between these two images comprising a test phase of several hypotheses homographic models of said movement by a RANSAC type algorithm operating on a set of first points (Pli) of the first image and corresponding first presumable points (P2i) of the second image so as to deliver the best homographic model hypothesis, this best homographic model hypothesis defining said motion, characterized in that said test phase comprises a test of several first homographic model hypotheses (Hlk) of said movement obtained from a set of second points ( Fold) of the first image and second suspected points corresponding (P2i) of the second image and at least one second homographic model hypothesis (H2) obtained from auxiliary information (Ox, Oy, Oz) provided by at least one inertial sensor (30) and representative of a movement of said sensor images between the captures of the two successive images (IM1, IM2) of said pair.
    [0002]
    The method of claim 1, wherein said auxiliary information is provided by at least one gyroscope (30).
    [0003]
    3. The method of claim 2, wherein said auxiliary information is provided by a gyroscope (30) and at least one other sensor taken from the group consisting of one or more accelerometers (31) and a magnetometer (32).
    [0004]
    4. Method according to one of the preceding claims, wherein using a preemptive RANSAC type algorithm.
    [0005]
    The method according to one of the preceding claims, wherein the testing of each homographic model assumption takes into account a distance dk (Hlk, H2) between said homographic model hypothesis and said at least one second homographic model assumption. (H2).
    [0006]
    The method of claim 5, wherein the test of each homographic model assumption comprises for each first point (BL1A,) of at least one block of said set of first points of the first image, a determination of a first estimated point in said second image from the homographic model hypothesis tested, a determination of a positional difference between the first estimated point and said corresponding first corresponding point (BL2A,) in the second image, a determination of a first score information (SCV1k, 15 SCV2) from the positional deviations obtained and an error tolerance (ERR), and a correction (27) of said first score information with a corrective element (CORR) comprising a first coefficient (clk) taking into account said distance, so as to obtain a second score information (SCV1Ck, SCV2C), this second score information being used for the determination of the adite better hypothesis of homographic model.
    [0007]
    7. The method according to claim 6, wherein the determination of said corrective element also comprises a weighting of said first coefficient (clk) by a weighting coefficient (X) representative of a weight of the first score information associated with said at least one a second homographic model hypothesis with respect to the first score information associated with said tested homographic model hypothesis. 30
    [0008]
    The method of claim 7, wherein the weighting coefficient (X) has a fixed and identical value for all tested homographic model assumptions of all the pairs of images. 3027144 24
    [0009]
    The method of claim 7, wherein the weighting coefficient (X) has a fixed and identical value for all tested homographic model assumptions of said current pair of images, which value is calculated from all values respective distances between the tested homographic model hypotheses of said current image pair and the second homographic model hypothesis, this value (Xp) being recalculated to each new current pair of images.
    [0010]
    10. Method according to one of claims 6 to 9, wherein the determination of said corrective element also takes into account the number (N) of second points.
    [0011]
    11. Apparatus for determining motion between successive video images, comprising input means (10) configured to receive image signals relating to video images successively captured by an image sensor, means (10) processors configured to perform, for each current pair of first and second successive video images, a motion determination between these two images, the processing means comprising test means (100) configured to test several hypotheses of homographic models of said movement by a RANSAC type algorithm operating on a set of first points of the first image and corresponding first presumed points of the second image so as to deliver the best homographic model hypothesis, this best homographic model hypothesis defining said motion, characterized in that it comprises auxiliary input means (11) configured to receive auxiliary information from at least one inertial sensor (30) and representative of a movement of said image sensor between captures of two successive images of said pair, the test means (100) being configured to test several first hypotheses of homographic models of said movement obtained from a set of second points of the first image and second corresponding presumed points of the second image and at least one second hypothesis of a homographic model obtained from said auxiliary information.
    [0012]
    The apparatus of claim 11, wherein the auxiliary input means (11) is configured to receive said auxiliary information from at least one gyroscope (30).
    [0013]
    The device according to claim 12, wherein the auxiliary input means (11) is configured to receive said auxiliary information from a gyroscope (30) and at least one other sensor taken from the group formed by one or more accelerometers 10 (31) and a magnetometer (32).
    [0014]
    14. Device according to one of claims 11 to 13, wherein the RANSAC type algorithm is a preemptive RANSAC type algorithm.
    [0015]
    15. Device according to one of claims 11 to 14, wherein the test means (100) are further configured for, when testing each homographic model hypothesis, take into account a distance dk (Hlk, H2) between said homographic model hypothesis and said at least one second homographic model hypothesis. 20
    [0016]
    Apparatus according to claim 15, wherein the test means comprises a test module (100) configured to test each tested homographic model hypothesis, said test module comprising first determining means (1001) configured to determine, for each first point of at least one block of said set of first points of the first image, a first estimated point in said second image from the tested homographic model hypothesis, second determination means (1002) configured for determining a positional difference between the first estimated point and said first corresponding presumed point in the second image, 3027144 26 of the third determination means (1003) configured to determine a first score information from the positional deviations obtained and a error tolerance, calculating means (1004) configured to calculate a patch comprising a first coefficient taking into account said distance, and correction means (1005) configured to perform a correction of said first score information with said corrective element so as to obtain a second score information, said second score information being used to determining said best homographic model hypothesis.
    [0017]
    The apparatus of claim 16, wherein the calculating means (1004) is further configured to weight said first coefficient by a weighting coefficient representative of a weight of the first score information associated with said at least one a second homographic model hypothesis with respect to the first score information associated with said tested homographic model hypothesis. 20
    [0018]
    The device of claim 17, wherein the weighting coefficient (X) has a fixed and identical value for all homographic model assumptions tested of all the pairs of images.
    [0019]
    19. Apparatus according to claim 17, wherein the weighting coefficient (Xp) has a fixed and identical value for all tested homographic model assumptions of said current pair of images, the calculating means being configured to calculate this value. from all the respective distance values between the tested homographic model assumptions of said current pair of images and the second homographic model hypothesis, and to recalculate this value with each new current pair of images. 3027144 27
    [0020]
    20. Device according to one of claims 16 to 19, wherein the calculation means (1004) are further configured to also take into account the number of second points.
    [0021]
    21. Processing unit, for example a microprocessor, incorporating a device according to one of claims 11 to 20.
    [0022]
    Apparatus, for example a cellular mobile phone or digital tablet, incorporating an image sensor (12), at least one inertial sensor (30), and a processing unit (1) according to claim 21, coupled to said sensor at least one inertial sensor, so as to receive said image signals and said auxiliary information.
    类似技术:
    公开号 | 公开日 | 专利标题
    FR3027144A1|2016-04-15|METHOD AND DEVICE FOR DETERMINING MOVEMENT BETWEEN SUCCESSIVE VIDEO IMAGES
    US20180240265A1|2018-08-23|Systems and Methods for Depth-Assisted Perspective Distortion Correction
    US9444991B2|2016-09-13|Robust layered light-field rendering
    EP2715662B1|2019-07-17|Method for localisation of a camera and 3d reconstruction in a partially known environment
    US7929801B2|2011-04-19|Depth information for auto focus using two pictures and two-dimensional Gaussian scale space theory
    US11244464B2|2022-02-08|Method and apparatus for performing depth estimation of object
    TW201118791A|2011-06-01|System and method for obtaining camera parameters from a plurality of images, and computer program products thereof
    WO2016164166A1|2016-10-13|Automated generation of panning shots
    FR2984057A1|2013-06-14|VIDEO FILM TURNING SYSTEM
    US8903124B2|2014-12-02|Object learning method, object tracking method using the same, and object learning and tracking system
    US10187584B2|2019-01-22|Dynamic range extension to produce high dynamic range images
    US20190304161A1|2019-10-03|Dynamic real-time texture alignment for 3d models
    US20190073787A1|2019-03-07|Combining sparse two-dimensional | and dense three-dimensional | tracking
    FR3007175A1|2014-12-19|TURNING CAMERA POSITIONING SYSTEMS FOR TURNING VIDEO FILMS
    US20140369622A1|2014-12-18|Image completion based on patch offset statistics
    WO2020152521A1|2020-07-30|Systems and methods for transforming raw sensor data captured in low-light conditions to well-exposed images using neural network architectures
    FR3053817A1|2018-01-12|METHOD AND SYSTEM FOR RECONSTRUCTING THREE-DIMENSIONAL REPRESENTATION
    FR3043235A1|2017-05-05|CALIBRATION OF A VIDEO MAPPING SYSTEM
    WO2019081587A1|2019-05-02|Image restoration method
    FR3051066A1|2017-11-10|METHOD FOR RESTORING IMAGES
    FR2888346A1|2007-01-12|Stereoscopic image sequence obtaining method for e.g. digital camera, involves obtaining mapped image by applying calculated transformation to image of monoscopic image pair and storing image in location chosen based on reference
    WO2020127774A1|2020-06-25|Device and method for estimating the movement of an image sensor between two images, and associated computer program
    JP2019512781A|2019-05-16|Method for reconstructing 3D multi-viewpoint by feature tracking and model registration.
    EP3072110A1|2016-09-28|Method for estimating the movement of an object
    WO2016075413A1|2016-05-19|Device for generating cinematographic rushes via video analysis
    同族专利:
    公开号 | 公开日
    US9838572B2|2017-12-05|
    FR3027144B1|2016-11-04|
    US20160105590A1|2016-04-14|
    引用文献:
    公开号 | 申请日 | 公开日 | 申请人 | 专利标题
    US10225473B2|2015-12-17|2019-03-05|Stmicroelectronics Sa|Threshold determination in a RANSAC algorithm|
    US10229508B2|2015-12-17|2019-03-12|Stmicroelectronics Sa|Dynamic particle filter parameterization|
    US10268929B2|2015-12-17|2019-04-23|Stmicroelectronics Sa|Method and device for generating binary descriptors in video frames|
    US10395383B2|2015-12-17|2019-08-27|Stmicroelectronics Sa|Method, device and apparatus to estimate an ego-motion of a video apparatus in a SLAM type algorithm|CA2687913A1|2009-03-10|2010-09-10|Her Majesty The Queen In Right Of Canada, As Represented By The Minister Of Industry Through The Communications Research Centre Canada|Estimation of image relations from point correspondences between images|
    US8798387B2|2010-11-11|2014-08-05|Panasonic Intellectual Property Corporation Of America|Image processing device, image processing method, and program for image processing|
    US8964041B2|2011-04-07|2015-02-24|Fr Vision Ab|System and method for video stabilization of rolling shutter cameras|US9374532B2|2013-03-15|2016-06-21|Google Inc.|Cascaded camera motion estimation, rolling shutter detection, and camera shake detection for video stabilization|
    US9277129B2|2013-06-07|2016-03-01|Apple Inc.|Robust image feature based video stabilization and smoothing|
    法律状态:
    2015-09-28| PLFP| Fee payment|Year of fee payment: 2 |
    2016-04-15| PLSC| Search report ready|Effective date: 20160415 |
    2016-09-21| PLFP| Fee payment|Year of fee payment: 3 |
    2017-09-21| PLFP| Fee payment|Year of fee payment: 4 |
    2018-09-19| PLFP| Fee payment|Year of fee payment: 5 |
    2019-09-19| PLFP| Fee payment|Year of fee payment: 6 |
    2020-09-17| PLFP| Fee payment|Year of fee payment: 7 |
    2021-09-22| PLFP| Fee payment|Year of fee payment: 8 |
    优先权:
    申请号 | 申请日 | 专利标题
    FR1459675A|FR3027144B1|2014-10-09|2014-10-09|METHOD AND DEVICE FOR DETERMINING MOVEMENT BETWEEN SUCCESSIVE VIDEO IMAGES|FR1459675A| FR3027144B1|2014-10-09|2014-10-09|METHOD AND DEVICE FOR DETERMINING MOVEMENT BETWEEN SUCCESSIVE VIDEO IMAGES|
    US14/848,962| US9838572B2|2014-10-09|2015-09-09|Method and device for determining movement between successive video images|
    [返回顶部]